Skip to content

Adding dependency on google-cloud-bigquery-storage temporarily for google #49807

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
potiuk merged 2 commits into from
Apr 26, 2025
Merged

Adding dependency on google-cloud-bigquery-storage temporarily for google #49807

potiuk merged 2 commits into from
Apr 26, 2025

Conversation

@amoghrajesh
Copy link
Contributor

@amoghrajesh amoghrajesh commented Apr 26, 2025

Due to: #49797, we conditionally add dependency on ray and this now introduces an issue where google-cloud-bigquery-storage doesnt install for python 3.12

Python 3.12:

31c31
< PyAthena==3.12.2
---
> PyAthena==3.13.0
49c49
< aenum==3.1.15
---
> aenum==3.1.16
52a53
> aiohttp-cors==0.8.1
138c139
< celery==5.5.1
---
> celery==5.5.2
142c143
< cfn-lint==1.34.1
---
> cfn-lint==1.34.2
154a156
> colorful==0.5.6
212c214
< google-api-python-client==2.167.0
---
> google-api-python-client==2.168.0
275c277
< h11==0.14.0
---
> h11==0.16.0
280c282
< httpcore==1.0.8
---
> httpcore==1.0.9
306c308
< ipython==9.1.0
---
> ipython==9.2.0
367c369,370
< msal==1.32.0
---
> msal==1.32.3
> msgpack==1.1.0
375c378
< mypy-boto3-rds==1.38.0
---
> mypy-boto3-rds==1.38.2
396,398c399,403
< openlineage-integration-common==1.31.0
< openlineage-python==1.31.0
< openlineage_sql==1.31.0
---
> opencensus-context==0.1.3
> opencensus==0.11.4
> openlineage-integration-common==1.32.0
> openlineage-python==1.32.0
> openlineage_sql==1.32.0
455a461
> py-spy==0.4.0
458c464
< pyarrow-hotfix==0.6
---
> pyarrow-hotfix==0.7
508c514,515
< qdrant-client==1.14.1
---
> qdrant-client==1.14.2
> ray==2.42.0
554a562
> smart-open==7.1.0
593c601
< structlog==25.2.0
---
> structlog==25.3.0
599c607
< teradatasql==20.0.0.29
---
> teradatasql==20.0.0.30
671c679
< ydb==3.21.0
---
> ydb==3.21.1

Python 3.11:

53c53
< PyAthena==3.12.2
---
> PyAthena==3.13.0
71c71
< aenum==3.1.15
---
> aenum==3.1.16
74a75
> aiohttp-cors==0.8.1
255,256c256,257
< celery==5.5.1
< certifi==2025.1.31
---
> celery==5.5.2
> certifi==2025.4.26
259c260
< cfn-lint==1.34.1
---
> cfn-lint==1.34.2
271a273
> colorful==0.5.6
331c333
< google-api-python-client==2.167.0
---
> google-api-python-client==2.168.0
341a344
> google-cloud-bigquery-storage==2.31.0
394c397
< h11==0.14.0
---
> h11==0.16.0
399c402
< httpcore==1.0.8
---
> httpcore==1.0.9
484c487,488
< msal==1.32.0
---
> msal==1.32.3
> msgpack==1.1.0
492c496
< mypy-boto3-rds==1.38.0
---
> mypy-boto3-rds==1.38.2
512a517,518
> opencensus-context==0.1.3
> opencensus==0.11.4
573a580
> py-spy==0.4.0
576c583
< pyarrow-hotfix==0.6
---
> pyarrow-hotfix==0.7
627c634,635
< qdrant-client==1.14.1
---
> qdrant-client==1.14.2
> ray==2.42.0
671a680
> smart-open==7.1.0
677c686
< snowflake-snowpark-python==1.30.0
---
> snowflake-snowpark-python==1.31.0
712c721
< structlog==25.2.0
---
> structlog==25.3.0
718c727
< teradatasql==20.0.0.29
---
> teradatasql==20.0.0.30
768c777
< uv==0.6.16
---
> uv==0.6.17
791c800
< ydb==3.21.0
---
> ydb==3.21.1

Leading to CI failures like: https://github.com/apache/airflow/actions/runs/14676573280

Temp attaching a new dependency to fix ci

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Apr 26, 2025
@amoghrajesh amoghrajesh added the full tests needed We need to run full set of tests for this PR to merge label Apr 26, 2025
@amoghrajesh amoghrajesh reopened this Apr 26, 2025
@amoghrajesh
Copy link
Contributor Author

amoghrajesh commented Apr 26, 2025

I just want to run the full set of tests on this one to be sure

@amoghrajesh amoghrajesh changed the title Adding dependency on google-cloud-bigquery-storage temporarily for google Adding dependency on google-cloud-bigquery-storage temporarily for google Apr 26, 2025
@potiuk
Copy link
Member

potiuk commented Apr 26, 2025

3rd-time is a charm.

@potiuk potiuk added the all versions If set, the CI build will be forced to use all versions of Python/K8S/DBs label Apr 26, 2025
@potiuk
Copy link
Member

potiuk commented Apr 26, 2025 •
edited
Loading

If you want to run for all versions (we need to tests 3.12) you also need to add "all versions" label. Added and rebased to re-run.

@potiuk potiuk merged commit 859c32b into apache:main Apr 26, 2025
135 checks passed
@potiuk
Copy link
Member

potiuk commented Apr 26, 2025

Merged - > 3.12 tests passed.

jroachgolf84 pushed a commit to jroachgolf84/airflow that referenced this pull request Apr 30, 2025
@amoghrajesh @jroachgolf84
Adding dependency on google-cloud-bigquery-storage temporarily for go…
de25c3f 
…ogle (apache#49807)
@eladkal eladkal mentioned this pull request Jun 15, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@potiuk potiuk potiuk approved these changes

@gopidesupavan gopidesupavan Awaiting requested review from gopidesupavan

@eladkal eladkal Awaiting requested review from eladkal

Assignees

No one assigned

Labels

all versions If set, the CI build will be forced to use all versions of Python/K8S/DBs area:providers full tests needed We need to run full set of tests for this PR to merge provider:google Google (including GCP) related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@amoghrajesh @potiuk